<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />

  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  <title>Topology &mdash; SIMULATeQCD 0.7 documentation</title>
      <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
      <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
      <link rel="stylesheet" href="../_static/togglebutton.css" type="text/css" />
      <link rel="stylesheet" href="../_static/custom.css" type="text/css" />
  <!--[if lt IE 9]>
    <script src="../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
        <script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
        <script src="../_static/jquery.js"></script>
        <script src="../_static/underscore.js"></script>
        <script src="../_static/doctools.js"></script>
        <script src="../_static/togglebutton.js"></script>
        <script>var togglebuttonSelector = '.toggle, .admonition.dropdown';</script>
    <script src="../_static/js/theme.js"></script>
    <link rel="index" title="Index" href="../genindex.html" />
    <link rel="search" title="Search" href="../search.html" />
    <link rel="next" title="Input and output formats for gauge configurations" href="configurationIO.html" />
    <link rel="prev" title="Gaugefield" href="gaugefield.html" /> 
</head>

<body class="wy-body-for-nav"> 
  <div class="wy-grid-for-nav">
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #343131" >
            <a href="../index.html" class="icon icon-home"> SIMULATeQCD
          </a>
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>
        </div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../01_gettingStarted/gettingStarted.html">Getting started</a></li>
<li class="toctree-l1"><a class="reference internal" href="../02_contributions/contributions.html">Contributions</a></li>
<li class="toctree-l1"><a class="reference internal" href="../03_applications/applications.html">Applications</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="codeBase.html">Code base</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="indexer.html">Indexer</a></li>
<li class="toctree-l2"><a class="reference internal" href="fileWriter.html">File Writer</a></li>
<li class="toctree-l2"><a class="reference internal" href="latticeContainer.html">Lattice Container</a></li>
<li class="toctree-l2"><a class="reference internal" href="communicationBase.html">Communication</a></li>
<li class="toctree-l2"><a class="reference internal" href="functorSyntax.html">Functor syntax</a></li>
<li class="toctree-l2"><a class="reference internal" href="gaugefield.html">Gaugefield</a></li>
<li class="toctree-l2 current"><a class="current reference internal" href="#">Topology</a></li>
<li class="toctree-l2"><a class="reference internal" href="configurationIO.html">Input and output formats for gauge configurations</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../05_modules/modules.html">Modules</a></li>
</ul>

        </div>
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu"  style="background: #343131" >
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../index.html">SIMULATeQCD</a>
      </nav>

      <div class="wy-nav-content">
        <div class="rst-content">
          <div role="navigation" aria-label="Page navigation">
  <ul class="wy-breadcrumbs">
      <li><a href="../index.html" class="icon icon-home"></a> &raquo;</li>
          <li><a href="codeBase.html">Code base</a> &raquo;</li>
      <li>Topology</li>
      <li class="wy-breadcrumbs-aside">
            <a href="../_sources/04_codeBase/topology.md.txt" rel="nofollow"> View page source</a>
      </li>
  </ul>
  <hr/>
</div>
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
             
  <section class="tex2jax_ignore mathjax_ignore" id="topology">
<h1>Topology<a class="headerlink" href="#topology" title="Permalink to this headline"></a></h1>
<p>You want to distribute your MPI ranks according to your <em>nvlink</em> Topology on the respective node.
Usually the ideal mapping should be done automatically, however if this is not the case you can try
to improve it by using the topology parameter.
If you do not want to handle this manually, just set <code class="docutils literal notranslate"><span class="pre">Topology</span> <span class="pre">=</span> <span class="pre">0</span> <span class="pre">0</span> <span class="pre">0</span> <span class="pre">0</span></code> or do not specify it at all in your
parameter file.</p>
<section id="how-it-works">
<h2>How it works<a class="headerlink" href="#how-it-works" title="Permalink to this headline"></a></h2>
<p>With the topology parameter we can map the cartesian coordinate of an MPI rank to a device, we set</p>
<div class="highlight-C++ notranslate"><div class="highlight"><pre><span></span><span class="n">myInfo</span><span class="p">.</span><span class="n">deviceRank</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">myInfo</span><span class="p">.</span><span class="n">coord</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">Topo</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">myInfo</span><span class="p">.</span><span class="n">coord</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">Topo</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">myInfo</span><span class="p">.</span><span class="n">coord</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">Topo</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">myInfo</span><span class="p">.</span><span class="n">coord</span><span class="p">[</span><span class="mi">3</span><span class="p">]</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">Topo</span><span class="p">[</span><span class="mi">3</span><span class="p">];</span><span class="w"></span>
</pre></div>
</div>
<p>E.g. when you want to split your lattice in 8 parts, by e.g. setting <code class="docutils literal notranslate"><span class="pre">Nodes</span> <span class="pre">=</span> <span class="pre">1</span> <span class="pre">1</span> <span class="pre">4</span> <span class="pre">2</span></code> you can use <code class="docutils literal notranslate"><span class="pre">Topology</span> <span class="pre">=</span> <span class="pre">0</span> <span class="pre">0</span> <span class="pre">1</span> <span class="pre">4</span></code>
to get a “good” mapping of MPI ranks to deviceRanks.</p>
</section>
<section id="some-cluster-specific-recommendations">
<h2>Some cluster-specific recommendations<a class="headerlink" href="#some-cluster-specific-recommendations" title="Permalink to this headline"></a></h2>
<p>If you want to spilt by setting <code class="docutils literal notranslate"><span class="pre">Nodes</span> <span class="pre">=</span> <span class="pre">2</span> <span class="pre">2</span> <span class="pre">2</span> <span class="pre">1</span></code> the optimal topology is <code class="docutils literal notranslate"><span class="pre">Topology</span> <span class="pre">=</span> <span class="pre">1</span> <span class="pre">2</span> <span class="pre">4</span> <span class="pre">0</span></code>.
With this topology setting you distribute the ranks over
the corners of a cube, just like the <code class="docutils literal notranslate"><span class="pre">nvlink</span></code> Topology on the Bielefeld cluster.
If you want to split the lattice in 6 parts by setting <code class="docutils literal notranslate"><span class="pre">Nodes</span> <span class="pre">=</span> <span class="pre">1</span> <span class="pre">1</span> <span class="pre">2</span> <span class="pre">3</span></code> you can use the <code class="docutils literal notranslate"><span class="pre">Topology</span> <span class="pre">=</span> <span class="pre">0</span> <span class="pre">0</span> <span class="pre">4</span> <span class="pre">1</span></code>
to distribute 3 Ranks on 3 GPUs on the top face and three ranks on the 3 GPUs on the bottom face of cube spanned by
the <code class="docutils literal notranslate"><span class="pre">nvlink</span></code> topology on the Bielefeld cluster.</p>
<p>On Summit, where you only have 6 GPUs per node and 3 GPUs per socket, you would choose
<code class="docutils literal notranslate"><span class="pre">Nodes</span> <span class="pre">=</span> <span class="pre">1</span> <span class="pre">1</span> <span class="pre">2</span> <span class="pre">3</span></code> with <code class="docutils literal notranslate"><span class="pre">Topology</span> <span class="pre">=</span> <span class="pre">0</span> <span class="pre">0</span> <span class="pre">3</span> <span class="pre">1</span></code>.</p>
</section>
</section>


           </div>
          </div>
          <footer>

  <hr/>

  <div role="contentinfo">
    <p>&#169; Copyright 2021, LatticeQCD.</p>
  </div>

  Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
    <a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
    provided by <a href="https://readthedocs.org">Read the Docs</a>.
   

</footer>
        </div>
      </div>
    </section>
  </div>
  <script>
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script> 

</body>
</html>